Hereinafter, the present invention and its underlying problem is described with regard to the processing of a video signal for line based motion estimation and motion compensation within a video processing apparatus such as a microprocessor or microcontroller having line memory devices, whereas, it should be noted, that the present invention is not restricted to this application, but can also be used for other video processing apparatus.
The present invention relates to a motion estimation and compensation device and more particularly to a motion estimation and motion compensation device that estimates motion vectors and performs motion compensated predictions of an interlaced sequence of chrominance sub-sampled video frames.
In digital compression coding standards of the Moving Picture Experts Group (MPEG) are widely used today in these fields of for example DVD videos and digital TV broadcasting to record or transmit large amounts of motion image data of a high quality. Such video compression standards of this MPEG-family, such as the MPEG-1, MPEG-2, MPEG-4, H.261, H.263, H.264 standards divide an input image into several macroblocks. Each of these macroblocks is encoded in each of the interprediction and intraprediction encoding mode available the bit rates required for encoding the macroblock and the rate distortion (RD) costs of the various encoding modes are compared. Then an appropriate encoding mode is selected according to the result of comparison and the macroblock is encoded using the selected encoding mode.
Using interprediction, the motion of a macroblock of a current frame of a whole motion picture, which comprises several consecutive frames, is predicted by referring to a macroblock of a neighboring frame of the current frame to reduce temporal redundant components using similarities between the video frames. In interprediction motion estimation and motion compensation are performed in units of a macroblock.
Motion estimation involves searching a predetermined region or area (so-called search area) of a reference frame extending from a position corresponding to a current macroblock to select an area having the smallest distance with the current macroblock as a most similar area and outputting a special difference between the most similar area and the current macroblock as a motion vector. Motion compensation involves reading the most similar area from the reference frame using the motion vector obtained through motion estimation and generating an interpredicted block. In motion estimation, the current macroblock data and the search area data of the reference frame must be read from an internal memory device, such as an SRAM. Because the data of the current macroblock and the search area data are used repetitively in motion estimation of the current macroblock, the motion estimation device includes an internal memory device such as a synchronous random excess memory (SRAM) to read data required from the external memory device and stores the read data in the internal memory device for the use in motion estimation.
In motion estimation, there is in fact a large overlap between the search area data of adjacent macroblocks. Thus, the motion estimation device reads only the search area data of the next macroblock required for motion estimation, not the overlapping search area data. The search area data read from the external memory device is stored in an internal memory device. The internal memory device is divided into several internal memories due to data transmission with limitations of the internal memory device and for the use of overlapping search area data. For example, when a macroblock has a size of 16×16 a search area has a size of 144×80, the data transmission bandwith of the internal memory device is for example 128 bits and the number of bits per pixle is 8 bits, then the 144×80 pixels of the search area are stored in 9 16×80 internal memories. When the search area data is stored in divided internal memories, the total number of gates of the internal memory device is increased due to an increase in overhead and thus, the area occupied by the internal memory device also increases. Moreover, as the number of divided internal line memories increases, the number of data ports increases and the number of combination logic units for a multiplexing data read from each internal line memory also increases.
Therefore, there is a need for an efficient internal memory included in a motion estimation device to store data from a reference frame.
In present systems, image data of this search area or windows is usually stored in a local buffer or on chip memory, the so-called line memory, to which rather extreme bandwith requirements are made.
Motion estimation and motion compensation—or shortly MEMC—are well known technologies to accomplish temporal conversions of incoming video signals to be displayed on a screen on a higher display frequency. With this can version so-called movie judders can occur. Especially to compensate these movie judders, MEMC is often used to perform an up-conversion from the standardized 24-Hz used for movie signals up to the display frequency of e.g. 50-Hz or 60-Hz.
FIG. 1 shows a schematic motion portrayal for movie sequences or frames in a motion picture, said movie sequences or frames having a standardized 24-Hz frame rate.
In particular, in FIG. 1 a motion trajectory of a moving object in original frames n−1, n, n+1, and n+2 of a moving picture is visualized, wherein original frames have a standardized 24-Hz frame rate, and wherein the moving object is presented in FIG. 1 as a circle object 111, 112, 113, 114 with regard of the corresponding frames. The motion trajectory of the moving object 111, 112, 113, 114 in frames n−1, n, n+1, n+2 is supposed to be a straight line.
In order to perform the up-conversion it is necessary to have access on minimum of two temporal different fields or frames which can be classified according to its occurrence in time as previous frame and current frame. Depending on the desirable quality and concerning the incoming data format, the access to further temporal frames of the motion picture can be required in order to accomplish a sufficient interpolation performance regarding video quality.
FIG. 2 shows a pull-down mode 2:3 for 60-Hz up-conversion of a 24-Hz movie signal or motion picture to a 60-Hz display signal. The 24-Hz movie signal or motion picture corresponds the movie signal or motion picture of FIG. 1.
When performing an up-conversion, artifacts like large area flicker and line flicker may occur. In case of fast moving objects a strong blurring effect can be induced. Thus, the quality of the displayed image is reduced considerably.
Such a situation is presented in FIG. 2. The moving object 111, 112, 113, 114 of FIG. 1 is shown as reference. As mentioned, the motion trajectory of the moving object 111, 112, 113, 114 is supposed to be a straight line. However, due to artifacts and due to blurring effect the motion of the moving object 211, 212, 213, 214, 215, 216, 217 is not smooth as desired and as predefined by the reference 111, 112, 113, 114.
In order to avoid artifacts and the blurring effect, up-conversion methods have been proposed capable of performing a motion estimation and compensation of moving objects in the consecutive frames of a motion picture such that the artifacts and the blurring effect are overcome or compensated, respectively. In particular, motion estimation and compensation provides for detecting the moving parts of the frames and interpolating missing frames according to estimated motion.
FIG. 3 shows a full motion compensated by up-conversion of a 24-Hz movie signal or motion picture of FIG. 1 to a 60-Hz display signal or motion picture.
In FIG. 3, due to motion estimation and motion compensation performed, the moving objects 211, 212, 213, 214, 215, 216, 217 provide the desired motion trajectory of FIG. 1 presented exemplary by the moving objects 111 and 113 of FIG. 1.
Commonly used algorithms compensate the motion in two directions, i.e. as well in the horizontal and the vertical direction. For that operation a memory access should be randomly possible, which requires for an application in hardware sufficient embedded chip memory within video processor for the different temporal incoming data streams. The size of this embedded chip memory strongly depends on the search range (search area) for the motion, as already outlined above, where the motion estimation can match similar video patterns in two temporal positions and derive the velocity of the motion in terms of pixels per frame or field.
However, this matching process does not always work perfectly, since methods to determine the quality of the measured vector are required. Therefore, for the internal storage of further temporal incoming video signals additional memory resources are required. This, however, increases the amount of embedded memory even further, which leads to a rather expensive implementation and thus to an increase of the chip size since for an integrated circuit it is the chip internal memory which significantly determines the chip area. Consequently, the chip is getting more expensive. Especially in the mainstream market segment such as for LCD-TVs these additional costs typically form a limiting factor for an implementation of a motion estimation and motion compensation method.
The present invention is, therefore, based on the object to provide a more efficient motion estimation and motion compensation, in particular a vertical motion estimation and vertical motion compensation, by which also a more efficient use of the chip-internal resources and especially of the chip-internal memory with regard to motion estimation and motion compensation can be enabled.